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1.  Introduction 


Welcome  to  the  INCOSE  Systems  Engineering  Measurement  Primer.  This  primer  is  not 
intended  to  be  a  tool,  a  case  study,  or  a  guideline.  It  is  a  basic  introduction  to 
measurement  for  the  beginning  measurement  practitioner  in  systems  engineering.  It  is 
written  from  the  perspective  that  the  reader  is  not  familiar  with  measurement  jargon,  does 
not  know  the  general  philosophies  shared  by  many  measurement  practitioners,  or  has  no 
experience  selecting,  specifying,  and  using  measures.  This  Primer  will  provide  the  reader 
with  the  basic  knowledge  of  measurement  from  which  more  specialized  topics  in 
measurement  can  be  explored. 

1.1  Organization  and  Objectives  of  the  Primer 

This  document  is  organized  to  address  two  main  goals.  The  first  objective  is  to  define  the 
basic  concepts  behind  measurement  and  measurement  programs  in  such  a  way  that  they 
will  be  usable  and  readable  by  anyone  regardless  of  their  experience  and  background; 
these  concepts  are  described  in  sections  2  and  3.  The  second  objective  is  to  provide  the 
background  knowledge  needed  to  prepare  you  to  set  up  a  measurement  program;  sections 
4  through  7  are  designed  to  achieve  this  aim. 

Section  8  is  intended  to  provide  feedback  on  the  primer.  Questions  or  comments  should 
be  submitted  per  the  instructions  in  this  section. 

1.2  Scope 

What  is  included  in  the  Primer:  The  fundamental  measurement  concepts  covered  in  this 
Primer  include  a  glossary  of  the  language  common  to  all  measurement  practitioners,  a 
description  of  the  principles  shared  by  most  measurement  programs  and  their  users,  and 
generic  components  of  successful  measurement  programs.  The  principles  included  here 
are  consistent  with  the  INCOSE  Metrics  Guidebook  for  Integrated  Systems  and  Product 
Development  and  the  Joint  Logistics  Commanders  (JLC)  Practical  Software  Measurement 
(PSM)  guidebook,  which  are  among  the  leading  guidebooks  for  systems  and  software 
measurement.  The  contents  of  this  Primer  also  encompass  a  proven  measurement  process 
(see  figure  1)  and  some  guiding  principles  for  those  intending  either  to  use  measures  or  to 
set  up  a  measurement  program.  This  discussion  provides  guidance  on  how  to  effectively 
use  measurement,  avoid  its  misuse,  select  good  measures,  obtain  the  benefits  from  correct 
use  of  measurement,  and  find  references  to  other  resources  that  discuss  more  specialized 
topics  in  measurement. 

Boundary  of  the  document:  This  Primer  is  not  a  comprehensive  guidebook;  it  does  not 
define  a  step  by  step  process  on  how  to  set  up  a  measurement  program.  It  is  not  directed 
toward  any  particular  organization  or  individual,  nor  is  it  an  identification  of  the  all¬ 
purpose  set  of  measures.  Most  importantly,  this  Primer  does  not  guarantee  the  success  of 
a  measurement  program  or  the  project  it  is  supporting. 
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Successful  measurement  programs  are  also  dependent  on  the  practitioners  having  a  good 
understanding  of  statistical  concepts  before  endeavoring  to  create  a  measurement 
program.  Many  of  the  terms  and  theories  commonly  used  by  measurement  practitioners 
are  derived  from  statistics.  A  few  statistical  concepts  are  mentioned  or  described  briefly  in 
this  Primer  for  informational  purposes  only;  however,  this  Primer  will  not  provide  the 
reader  with  any  detailed  explanations.  It  is  possible  to  create  a  successful  measurement 
program  without  having  a  solid  statistical  foundation,  but  it  may  make  it  more  difficult. 
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2.  Definitions  and  Commonly  Used  Terms 


The  list  below  defines  basic  measurement  terms  used  in  this  Primer. 

Causal  Analysis  A  systematic  method  for  identifying  specific  problem  areas  in  work 
products,  project  progress,  and  processes,  determining  the  causes  of 
problem  areas,  and  developing  and  implementing  solutions  to 
prevent  the  problem  areas  from  occurring  in  the  future. 

Control  Chart  A  graphical  method  for  evaluating  whether  a  process  is  or  is  not  in  a 

state  of  statistical  control  (SPC).  This  chart  analyzes  a  process 
attribute  measure  as  a  function  of  time  for  determining  process 
performance  status.  The  values  plotted  on  a  control  chart  are 
obtained  from  a  sequence  of  individual  measurements  for  any 
statistic. 

Control  Limits  (Upper  and  Lower)  The  upper  and  lower  values  of  a  process 

attribute  between  which  nearly  all  sample  points  fall.  These  control 
limits  form  a  tolerance  band  within  which  the  performance  of  the 
process  is  considered  to  be  statistically  in-control.  Control  limits  are 
often  represented  by  lines  on  a  control  chart  that  are  used  to  judge 
whether  the  process  performance  being  measured  is  out-of-control. 
For  example,  an  upper  control  limit  would  be  a  set  value  of  a 
measure  or  indicator  that  represents  a  boundary  inside  which 
accepted  values  of  the  measure  should  lie.  If  the  observed  value(s) 
exceeded  this  set  value,  or  upper  control  limit,  the 
process/performance  parameter  being  measured  would  be 
considered  “out  of  control.” 

Data  Element  Data  at  the  level  it  is  collected  and  prior  to  any  processing  or 

analysis. 

Demonstrated  The  parameter  value  that  is  demonstrated  or  estimated  through 

Value  analysis  and  modeling  or  measured  in  a  particular  test.  (Adapted 

from  DSMC  SEMG.) 


3 


Indicator  (Metric) 


Issue 

Linear  Regression 

Measure 

Measurement 


Measurement 

Analysis 


Measurement 

Program 


1)  A  measure  or  combination  of  measures  that  provides  insight  into 
an  issue  or  concept.  Indicators  are  often  comparisons,  such  as 
planned  versus  actual  measures,  which  are  usually  presented  as 
graphs  or  tables.  Indicators  can  describe  the  current  situation 
(current  indicators)  or  predict  the  future  situation  (leading 
indicators)  with  respect  to  an  issue.  (Adapted  from  PSM.) 

2)  A  mathematical  composite  of  relevant,  quantifiable,  product, 
project  progress  or  process  attributes  (measures)  taken  over  time 
that  communicate  important  information  about  quality,  processes, 
technology,  products,  projects,  and/or  resources. 

A  risk,  constraint,  objective,  or  concern,  often  associated  with 
resources,  progress,  quality,  or  performance.  Issues  represent 
current  or  potential  problem  areas  that  should  be  monitored.  (PSM) 

A  straight  line  representing  the  estimated  linear  relationship  between 
two  variables. 

The  result  of  counting  or  otherwise  quantifying  an  attribute  of  a 
process,  project  or  product.  Measures  are  numerical  values 
assigned  to  attributes  according  to  defined  criteria.  The  raw  data 
from  which  indicators  are  calculated.  Some  examples  of  these  are 
size,  cost,  and  defects.  (Synonymous  with  ‘measure’  is  a 
measurable ,  a  data  element,  or  a  primitive.) 

The  process  of  assigning  numerical  values  to  process,  product,  or 
project  attributes  according  to  defined  criteria.  This  process  can  be 
based  on  estimation  or  direct  measurement.  Estimation  results  in 
planned  or  expected  measures.  Direct  measurement  results  in  actual 
measures. 

The  use  of  measurement  data  to  identify  problems,  assess  problem 
impact,  project  outcomes,  and  evaluate  alternatives  related  to 
identified  issues.  Examples  of  measurement  analysis  are  estimation, 
feasibility  analysis,  and  performance  analysis.  (Adapted  from  PSM.) 

An  organizational  initiative  responsible  for  the  planning,  educating, 
and  facilitation/execution  of  measurement  for  the  purpose  of 
process,  product,  and/or  project  control  and  improvement. 
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Measurement  Tool 


Normalization 


Planned  Value 
(Target  or 
Expected  Value) 


Process  Measure 


Product  Measure 


Progress  Measure 


Root  Cause 
Analysis 


A  device  or  software  program  that  automates  some  process  (data 
collection,  measure  or  indicator  calculation,  graph  plotting,  etc.) 
within  a  measurement  program. 

A  technique  that  meaningfully  compare  or  combine  data  from 
different  processes,  products  or  projects,  or  with  different  units  of 
measurement.  This  often  requires  the  definition  and  validation  of 
conversion  rules  and/or  models.  For  example,  to  compare  the 
quality  of  work  produced  in  two  programs,  it  would  be  necessary  to 
look  at  defect  counts  in  relation  to  the  amount  or  size  of  the  work 
produced  in  the  same  units  of  measurement.  (Adapted  from  PSM.) 

The  anticipated  value  of  a  parameter  or  measure  at  a  given  point  in 
the  development  cycle.  The  anticipated  value  may  be  the  result  of 
historical  measurement  data,  an  empirical  model,  or  an  estimation 
tool.  A  plot  of  planned  value  versus  time  is  known  as  the  planned 
value  profile.  The  range  of  acceptable  values  is  called  the  tolerance 
band.  (Adapted  from  DSMC  SEMG.) 

A  measure  of  how  well  a  given  process  or  activity  is  working,  which 
can  provide  insight  into  process  stability  and  process  improvement 
opportunities.  Historical  data  from  process  measures  also  provides  a 
basis  for  estimation  of  processes  applied  on  similar  projects.  An 
example  of  this  type  of  measure  might  be  activity  cycle  time  or 
rework  factors. 

A  measure  of  the  characteristics  or  quality  of  an  end  item.  This  end 
item  could  be  anything  from  the  shipped  product  to  the  system 
design  specification  document  to  a  quantifiable  measure  of  service 
performed  or  level-of-effort  provided. 

This  measure  provides  project  or  program  status  in  terms  of 
schedule  and  cost.  It  measures  values  or  changes  over  time, 
generally  against  planned  values.  Most  budget/cost  and  schedule 
measures  are  progress  measures. 

The  process  by  which  a  single  event  that  represents  a  fault  in  work 
products  and  processes  is  analyzed  to  determine  the  fundamental 
cause  for  the  fault.  Based  on  this  understanding,  a  correction  can  be 
made  to  the  product  or  process  to  resolve  the  fault. 
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Statistical  Process 
Control  (SPC) 


(See  Section  6.5  for 
references  on  this 
topic.) 


Technical 

Performance 

Measure 


Variance 


1)  The  use  of  data  and  statistical  analysis  techniques  (e.g.,  control 
charts)  to  analyze  and  measure  variations  in  processes  in  order  to 
determine  whether: 

•  The  process  is  out-of-control  (if  a  problem  exists)  and  the  action 
required  to  correct  the  problem  (i.e.,  sources  of  variability  also 
called  assignable  causes) 

•  Improvement  actions  have  been  effective 

2)  SPC  is  a  method  for  monitoring,  controlling,  and  improving  a 
process  for  the  purpose  of  maintaining  the  performance  of  the 
process.  (Software  Productivity  Consortium) 

An  attribute  of  a  system  that  can  be  measured  to  determine  how 
well  a  system  is  satisfying  or  meeting  a  technical  requirement  or 
goal.  It  provides  an  assessment  of  the  product  design  by  estimating 
the  values  of  essential  performance  parameters  of  the  design 
through  engineering  analyses  and  tests.  TPMs  are  used  to: 

•  Forecast  the  values  to  be  achieved  through  the  planned  technical 
effort 

•  Measure  differences  between  the  achieved  values  and  those 
allocated  to  the  product  by  the  systems  engineering  process 

•  Determine  the  impact  of  these  differences  on  system 
effectiveness. 

The  purpose  of  a  TPM  is  to: 

•  Provide  visibility  of  actual  versus  planned  performance 

•  Provide  early  detection  or  prediction  of  problems  requiring 
management  attention 

•  Support  assessment  of  technical  impact  of  proposed  change 
alternatives. 

(Adapted  from  DSMC  SEMG.) 

1)  The  difference  between  the  planned  value  and  the  actual, 
demonstrated  or  current  value  of  a  measure  or  parameter. 

2)  A  measure  of  the  variability  of  a  random  variable,  calculated 
as  the  standard  deviation  squared. 
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3.  Measurement  Process  and  Supporting  Infrastructure 


Measurement  is  much  more  than  the  data  collected  and  the  calculation  of  a  indicator. 
Effective  measurement  requires  planning  of  what  will  be  measured,  how  the  measurement 
will  be  performed,  how  the  data  will  be  analyzed,  what  reporting  is  needed,  what  actions 
will  be  taken  for  the  results,  and  who  is  responsible  for  each  of  these  activities.  Thus,  a 
measurement  program  is  usually  established  that  defines  the  measurement  process  for  the 
organization  and  provides  an  infrastructure  to  support  the  measurement  process  activities. 

The  first  task  of  the  measurement  program  is  to  identify  its  purpose  for  measurement. 
Organizations  usually  perform  measurement  for  one  of  the  following  reasons  [SEI/MPM]: 

•  Characterize  or  gain  an  understanding  of  their  processes,  project  progress  and/or  products  and 
establish  baselines  for  future  assessment  of  these 

•  Evaluate  or  determine  project  progress  with  respect  to  plans 

•  Predict  resources,  schedule  and  performance  to  support  planning  and  trades 

•  Identify  improvement  opportunities  for  progress,  processes  and/or  products,  such  as 
roadblocks  to  progress,  root  causes  of  problems  in  products,  and  inefficiencies  in  processes 

These  will  be  discussed  in  more  depth  later  in  this  section. 

Once  the  purpose  for  performing  measurement  has  been  established,  it  is  essential  to 
determine  the  measurement  process  that  will  be  employed  by  the  organization.  Both  the 
purpose  and  process  for  measurement  needs  to  be  communicated  throughout  the 
organization  to  all  stakeholders.  The  definition  of  the  process  will  help  determine  what 
needs  to  be  included  in  the  supporting  infrastructure.  This  Primer  will  cover  common 
process  activities  and  practices  used  by  many  successful  measurement  programs  and  the 
typical  components  of  the  supporting  infrastructure. 

3.1  The  Measurement  Process 

Measurement  needs  to  be  viewed  as  a  process  for  obtained  vital  insight  into  the  progress, 
products,  and/or  processes  of  the  project  or  system  being  developed.  This  insight  helps 
the  decision  maker  to  make  more  informed  decisions,  identifying  deviations  from  plans 
earlier,  thus  allowing  mid-course  corrective  actions  when  it  is  least  expensive  to  make 
them.  Measurement  should  not  be  viewed  as  a  set  of  predefined  measures  that  never 
change  throughout  the  program.  Instead,  the  measures  used  need  to  address  the  current 
issues  at  hand,  which  may  change  with  time.  The  process  includes  the  activities  for 
selecting  and  specifying  the  measures,  establishing  a  measurement  plan,  planning  and 
executing  the  data  collection  and  storage,  analyzing  the  data,  reporting  the  results,  and 
most  importantly,  taking  action.  These  fundamental  activities  are  explained  in  this  section. 
Figure  1  shows  the  process  for  measurement  that  is  described  in  the  paragraphs  below. 
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The  measures  that  are  selected  are  the  key  to  successful  measurement.  Before  any  data  is 
collected,  the  project  management  team  and  stakeholders  perform  measurement  planning. 
This  includes  identifying  and  prioritizing  the  program  issues,  selecting  and  specifying 
appropriate  measures,  and  integrating  the  measurement  activities  into  the  standard 
processes  of  those  performing  the  work.  During  this  planning  activity,  current  and 
expected  issues  are  reviewed  to  identify  the  relevant  measures  (quantitative  data)  that, 
when  combined  with  other  program  data,  will  provide  insight  into  those  issues.  The 
objective  is  to  define  the  set  of  measures  that  provides  the  best  indicators  and  insight  for 
the  least  cost.  Thus,  measures  that  are  based  on  existing  data  sources  and/or  measurement 
tasks  should  be  given  special  consideration.  The  planning  may  need  to  be  revised  once  the 
development  or  maintenance  staff  is  identified  in  order  to  integrate  the  measures  into  their 
processes.  A  result  of  these  planning  tasks  is  the  creation  of  a  Measurement  Plan  for  the 
program. 

When  considering  which  measures  to  use,  the  attributes  described  in  this  section  should  be 
weighted  in  context  with  the  project’s  objectives,  constraints,  risks,  and  issues.  The 
measures  chosen  must  provide  insight  into  the  identified  objectives,  constraints,  risks,  and 


issues.  As  these  change,  the  measures  in  use  need  to  be  re-evaluated  for  adequacy  in 
providing  the  necessary  insight  and  changed,  if  necessary.  The  measure  remains  active 
only  if  it  has  a  valid,  justifiable  purpose.  Most  projects  can’t  afford  to  collect  data  and 
analyze  measures  that  will  never  be  used.  Therefore,  the  selection  (or  re-selection)  of 
measures  is  an  iterative  activity  that  occurs  throughout  the  life  of  the  project. 

3. 1.1.1  Attributes  of  Measures 

What  are  some  of  the  characteristics  of  measures  of  which  one  should  be  aware?  How 
does  one  know  if  a  measure  has  these  characteristics?  The  following  list  provides  a 
description  of  the  attributes  of  good  measures  as  well  as  questions  to  ask  to  determine  if  a 
measure  has  that  particular  attribute. 

•  Relevance.  “Why  do  I  want  to  collect  this  measure?  If  1  have  more  than  one  reason  for  having 
this  measure,  is  there  ambiguity  in  what  it  is  trying  to  accomplish?”  Only  select  measures  that 
do  not  have  numerous  interpretations  and  that  are  pertinent  to  an  end  result  you  are  frying  to 
obtain. 

•  Completeness.  “Have  I  covered  all  the  bases?  Have  1  left  out  a  key  parameter  that  is  needed 
to  analyze  my  results?  Is  there  a  need  to  weight  one  parameter  more  than  another?”  Be  sure 
you  identify  a  balanced  set  of  measures  and  that  your  emphasis  does  not  become  skewed. 

•  Timeliness.  “Did  I  find  out  what  I  needed  to  know  in  time  to  make  a  difference?”  Be  sure 
collection  and  analysis  will  provide  the  needed  information  in  time  to  allow  corrective  action  to 
be  initiated. 

•  Simplicity.  “Can  I  collect  and  analyze  the  data  easily  and  cost  effectively?  Can  the 
users/managers  understand  what  it  means?”  Keep  it  as  simple  and  logical  as  possible.  The 
measures  should  be  easy  to  collect,  analyze,  and  understand. 

•  Cost  Effectiveness.  “Can  I  afford  it?  Can  I  not  afford  it?  Does  it  provide  more  value  than  it 
costs?”  Use  data  that  is  economical  to  collect.  Use  organizational  or  customer  required  data 
to  address  other  program  issues,  where  applicable.  Leverage  data  collected  for  current 
management  practices. 

•  Repeatability.  “Will  the  same  conditions  provide  the  same  answer  twice?  Is  the  accuracy  and 
precision  adequate?”  This  is  important  for  comparing  measures  across  projects. 

•  Accuracy.  “Is  my  data  really  relevant  to  my  purpose?  Are  my  measures  reliable?  Am  I 
measuring  at  the  appropriate  time?”  Make  sure  that  your  measures  are  accurate  and  the 
resulting  analysis  accurately  serves  the  intended  puipose  of  the  measure. 

3. 1.1. 2  Identification  of  Project  Issues  (Objectives,  Risks,  Concerns,  and  Constraints ) 
Issue  identification  and  prioritization  is  the  first  activity  in  measurement  process.  Project 
issues  and  objectives,  which  vary  from  project  to  project,  drive  the  measurement 
requirements.  Thus,  there  is  no  “one  size  fits  all”  set  of  measures.  As  a  starting  point,  the 
project  management  team  and  stakeholders  identify  the  issues  specific  to  their  project  and 
prioritize  them. 

As  a  minimum,  the  following  should  be  inputs  to  the  issue  identification  task: 

•  Project  risk  analysis 

•  Project  constraints  and  assumptions 
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•  Product  acceptance  criteria 

•  Known  project  problems 

•  Project  goals  and  objectives 

•  External  requirements  or  dependencies 

The  result  of  this  task  is  a  prioritized  list  of  project  specific  issues,  sometimes  called  the 
Project  Issues  Report  (PIR).  The  prioritization  scheme  used  is  up  to  the  discretion  of  the 
project  management  team  and  applicable  stakeholders  and  is  usually  the  same  as  or  similar 
to  the  scheme  used  for  risk  prioritization  for  the  project. 

3. 1.1.3  Selecting  the  Measures 

There  are  many  methods  that  can  be  used  to  select  measures  for  a  project.  Two  proven 
methods  are  provided  in  the  following  sections. 

3.1. 1.3.1  Practical  Software/Systems  Measurement  (PSM)  Method 

Practical  Software  Measurement  (and  the  evolving  JLC/INCOSE  Practical  Systems 
Measurement)  guidance  includes  a  set  of  common  program  issues  (systems  issues  listed 
below  which  vary  slightly  from  PSM  guidebook)  that  affect  most  projects.  Specific 
project  domains  may  include  other  issues  not  considered  “common”.  The  common  project 
issues  are  as  follows: 

•  Schedule  and  Progress 

•  Resources  and  Cost 

•  System  Performance 

•  Growth  and  Stability 

•  Product  Quality 

•  Life  Cycle  Process 

•  Technology  Effectiveness  and  Adequacy 

•  Customer/User  Satisfaction 

An  organization  may  find  one  or  more  additional  “common  issues”  that  apply  for  all 
projects  in  their  business  domain.  Also,  relevant  issues  that  require  insight  will  change  as 
the  project  progresses.  These  “common  issues”  are  used  as  mechanisms  to  help  identify 
project  specific  issues  and  then  map  them  to  measurement  categories  and  appropriate 
candidate  measures  for  the  issue.  The  mechanism  descriptions  and  mapping  of  these 
mechanisms  are  documented  in  the  PSM  Guidebook  (see  references).  This  process 
includes: 

•  Identification  of  candidate  measures.  For  systems,  the  candidate  measures  may  include  some 
of  those  documented  in  the  PSM  guidebook  for  software,  where  they  pertain  to  project 
management  attributes  rather  than  product  attributes.  Other  sources  of  candidate  measures 
include  the  INCOSE  Guidebook,  various  standards,  and  texts  (see  references).  In  the  near 
future,  the  Practical  Systems  Measurement  Guidebook,  which  is  a  collaborative  project 
between  INCOSE  and  PSM,  will  also  contain  candidate  measures  for  engineering  a  system. 
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•  Establishment  of  selection  criteria.  Key  items  in  the  selected  criteria  include  the  measurement 
effectiveness  to  provide  the  desired  insight  into  the  issue,  the  applicability  of  the  measure  to  the 
project’s  domain,  the  compatibility  of  the  measure  to  the  current  management  practices,  the 
cost  and  availability  of  data  to  support  the  measure,  the  applicability  of  the  measure  to  the 
particular  life  cycle  phases,  and  how  it  addresses  any  external  measurement  requirements.  (See 
PSM,  section  2.3.2  for  guidance  on  selection  criteria.) 

•  Evaluation  of  candidate  measures  against  the  selection  criteria  to  select  the  “best”  measures 
(See  PSM,  section  2.3.2.) 

3.1. 1.3.2  GQM  Method 

Another  example  of  a  method  that  can  be  used  to  identify  appropriate  and  useful  measures 
from  identified  project  goals  is  the  Goal/Question/Metric  (GQM)  approach.  The  four 
basic  steps  of  GQM  are: 

1.  State  the  information  Goal.  Identify  the  information  consumer  groups  (stakeholders)  want  to 
know  and  determine  what  they  want  to  do  with  the  information.  Work  top-down,  including 
both  organizational  and  project  goals  as  appropriate. 

2.  Ask  the  Question.  What  questions  should  be  asked  to  determine  whether  the  goal  is  being  met? 

3.  Determine  the  Measure.  Identify  the  specific  parameters  that  must  be  measured  to  answer  the 
question(s)  posed  in  Step  2.  What  measures  are  needed  (directly  or  indirectly)  and  what  must 
be  measured  (collected)  to  obtain  it? 

4.  Do  and  evaluate.  Apply  the  measures  selected  and  evaluate  their  usefulness.  Return  to  Step  1 
or  2,  if  measures  are  inadequate  for  their  intended  purpose. 

3. 1.1. 4  Specification  of  the  Measures 

After  the  measures  are  selected,  it  is  important  to  specify  them  in  an  unambiguous  manner. 
The  measurement  specification  serves  as  the  common  set  of  instructions  for  obtaining, 
evaluating,  and  correctly  interpreting  the  measurement  data  for  a  specific  measure.  Thus, 
it  needs  to  mean  the  same  thing  to  each  practitioner.  The  specification  should  include: 

•  A  clear  definition  of  the  measure 

•  Data  types 

•  Data  collection  frequency  (on  a  periodic  basis,  not  event  driven  basis;  usually  monthly) 

•  Data  preparation  required 

•  Level  and  scope  of  measurement  (At  what  level  is  data  collected?  What  activities  is  it 
collected  from?) 

•  Applicable  phases  for  the  measure  and  data  collection 

•  Interpretation  notes 

The  result  of  this  task  is  a  set  of  measures  which  directly  address  the  program  issues,  are 
clearly  defined  and  documented  in  specifications  of  measures,  and  serve  as  a  basis  for 
integration  into  the  development  or  maintenance  process. 
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3.1.2  Data  Collection  Method 


A  clearly  defined  and  repeatable  process  needs  to  be  created  to  describe  the  method  by 
which  the  data  will  be  collected.  This  method  must  identify  the  point(s)  in  time  when  the 
data  will  be  collected,  what  (if  any)  tools  will  be  used  to  accomplish  collection,  the  people 
responsible  for  collecting  the  data,  how  the  data  will  be  validated,  and  what  is  done  with 
the  data  once  it  is  collected  (storage  and  preparation).  A  simple  example  of  a  data 
collection  method  could  be  that  on  the  first  day  of  every  month,  a  project  member  will  be 
responsible  for  manually  totaling  the  number  of  defects  found  in  a  particular  product  by 
any  customer  since  the  beginning  of  the  previous  month. 

The  measurement  analysis  results  that  will  be  performed  following  the  data  collection  can 
only  be  as  good  as  the  data  that  goes  into  the  analysis.  Therefore,  it  is  important  to 
validate  and  verify  the  data  that  is  collected.  To  help  ensure  that  data  collected  is  valid, 
the  following  should  be  attributes  of  the  data  collection  activities,  with  someone  given 
responsibility  to  ensure  the  action: 

•  All  contractors  and  subcontractors  should  participate  in  the  data  collection 

•  Delivered  data  is  periodically  audited  for  quality,  regardless  of  whether  the  data  is  delivered 
from  an  internal  or  external  source 

•  Determination  of  data  /  measure  combinations,  comparisons,  and  other  analysis  needed  is 
communicated  to  all  responsible  parties  (including  developers  or  maintainers) 

•  Data  collected  is  demonstrated  to  be  valid  by  the  collecting  party  and  is  verified  for  adequacy 


3.1.3  Calculation  Method:  Getting  from  Measures  to  Indicators 


In  some  instances,  a  raw  measure  is  the  indicator.  In  most  cases,  however,  the  indicator  is 
a  mathematical  combination  of  measures,  and  the  method  of  calculating  that  measure  or 
indicator  needs  to  be  defined  and  documented.  Most  indicators  compare  the  relationship 
of  two  variables  and  can  be  communicated  graphically.  For  each  issue,  there  should  be 
calculation  of  both  current  indicators  (assessing  the  current  situation),  such  as 
performance  with  respect  to  control  limits  and  conformance  to  plan  (variances  of  actual 
versus  plan),  and  leading  indicators  (predicting  the  future  situation),  such  as  issues  that 
may  become  problems,  questionable  trends,  and  feasibility  of  current  plans. 
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It  is  often  necessary  to  normalize  the  data  using  defined  conversion  rules  or  models  to 
support  meaningful  comparisons  between  different  project,  process,  or  product  attributes. 
These  rules  and  models  must  be  carefully  validated  with  historical  data.  The  measures  and 
indicators  then  need  to  be  aggregated  (combining  raw  data  and  lower  level  measures  into 
higher  level  summarizations)  to  a  level  adequate  for  the  decision  maker’s  needs. 
Aggregating  the  data  requires  defining  relationships  among  the  objects  that  are  measured. 

Where  feasible,  the  calculation  should  be  automated  through  supporting  technology.  Total 
number  of  defects/time  is  an  example  of  a  formula  definition  of  an  indicator.  The  data 
collected  is  the  number  of  defects  and  the  time  period  during  which  the  defects  were 
detected. 

3.1.4  Analysis  of  the  Measures  or  Indicators:  Measurement 
Interpretation 


To  optimize  program  and  system  performance,  careful  tradeoffs  must  be  made  among 
cost,  schedule,  quality,  and  functionality.  Appropriate  measures  and  indicators  (metrics) 
are  essential  inputs  to  this  tradeoff  analysis.  Periodic  analysis  of  the  relationships  between 
measurement  results  and  the  program  requirements  and  attributes  provides  insight  that 
helps  identify  problems  early,  when  they  are  relatively  inexpensive  to  resolve.  The  stored 
historic  measurement  data,  together  with  program  attribute  data,  form  the  basis  for 
predictive  models  that  should  be  used  to  estimate  cost,  schedule,  and  quality  elements  of 
the  project  or  product.  These  estimates  are  essential  for  realistic  and  justifiable  planning, 
change  impact  assessment,  and  tradeoff  decisions. 

Issue  analysis  is  a  systematic  analysis  process  in  which  indicators  are  analyzed  with 
qualitative  program  data  to  assess  program  status  and  risks  relative  to  known  issues.  The 
issue  analysis  includes  feasibility  analysis  to  determine  whether  the  plans  and  targets  are 
achievable  and  project  performance  analysis  to  determine  whether  those  plans  and  targets 
are  being  met.  The  indicators  and  performance  characteristics  are  examined  for  critical 
path  items  or  inconsistent  trends  that  may  be  potential  problems.  The  results  of  this 
analysis  are  then  used  to  determine  potential  corrective  actions  and  to  identify  new  issues. 

Within  the  analysis  activities,  variances  and  trends  in  data  are  the  key  to  identifying 
dependencies  among  measures,  as  well  as  departures  from  planned  values  of  the  measures. 
These  can  show  where  improvements  to  processes,  progress,  or  products  could  be  made. 
Variances,  which  are  a  comparison  between  planned  values  and  actual  measured  values, 
quickly  indicate  a  momentary  departure  from  the  plan  that  may  warrant  investigation  to 
prevent  further  digression.  However,  to  understand  a  trend,  and  there  may  be  a  need  to 
analyze  more  than  two  variables  at  one  time.  Specific  analysis  techniques  that  are 
identified  and  used  should  be  documented.  Any  process,  progress,  or  product  changes 
made  due  to  a  perceived  relationship  between  measures  should  be  reflected  in  the  analysis 
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of  future  data.  This  consistent  use  of  analysis  produces  more  predictable  results.  The 
analysis  results  should  always  be  validated  prior  to  preparing  a  presentation  of  the  results. 

3. 1.4.1  Goals/Control  Limits  for  Each  Measure 

Once  a  trend  in  the  measurement  data  has  been  identified,  even  if  the  trend  is  such  that  the 
value  of  the  measure  is  unpredictable  (i.e.,  processes  or  quality  of  products  are  not  under 
control),  there  may  be  a  desire  to  improve  that  trend.  Improvement  is  a  result  of  change, 
and  this  change  can  be  measured.  The  kind  of  change  desired  is  defined  by  the  planned  or 
target  value  of  the  measure.  If  the  value  of  the  measure  has  no  consistency  over  time,  as 
is  usually  the  case  when  the  processes  or  quality  of  the  products  are  not  under  control, 
then  there  may  be  a  chosen  range  of  values  that  are  considered  to  be  acceptable  or 
desirable  for  that  measure.  Over  time,  as  process  improvement  actions  bring  the  processes 
and  quality  come  under  control,  this  range  of  values  may  be  shortened  to  reflect  more 
stringent  acceptable  or  desirable  values  for  that  measure.  This  range  of  values,  called  the 
tolerance  band,  is  defined  by  upper  and  lower  control  limits.  If  the  measure  has  a 
predictable  value,  then  there  may  be  a  value  of  the  measure  that  is  defined  to  be  the 
MOST  acceptable  or  desirable.  This  value  then  becomes  the  goal  of  the  measure,  and 
changes  made  are  focused  on  driving  consistent  values  of  the  measure  closer  to  the  goal. 
Setting  goals  and  control  limits  for  measures  or  indicators  is  intrinsic  to  controlled 
improvement. 

3.1.5  Reporting  and  Using  the  Results 


The  results  of  measurement  analysis  must  feed  back  into  the  measurement  process.  Part 
of  the  measurement  program  documentation  needs  to  describe  how  results  will  be  used. 
The  use  of  the  results  is  the  most  important  step  of  the  process.  Improvement  of  the 
project,  process,  or  product  as  a  result  of  measurement  can  only  be  realized  through 
appropriate  follow-up  actions.  For  example,  results  can  be  used  to  orient  decisions  to  the 
best  alternative  or  to  obtain  understanding  of  causal  relationships  which,  in  turn,  facilitate 
corrective  actions  to  be  initiated.  (For  more  information  on  use  of  measurement,  see 
section  4,  Purposes  of  Measurement,  and  section  5,  Application  Guidance  and  Lessons 
Learned.) 

3.2  Measurement  Program  Supporting  Infrastructure 

An  environment  must  be  established  that  is  built  on  management  commitment  to  the 
objectives  of  the  measurement  program.  These  objectives  must  be  clearly  communicated 
to  all  stakeholders.  Understanding  the  meaning  of  the  selected  measures  and  indicators 
and  how  measurement  is  going  to  be  implemented  and  used  will  help  ensure  the 
acceptance  and  success  of  the  program.  Adequate  resources  must  be  allocated  to 
performing  the  measurement  activities  from  the  selection  of  measures  to  the  taking  of 
actions.  Appropriate  planning,  training  and  tools  are  other  necessary  components  of  the 
infrastructure.  The  key  to  successful  acceptance,  support,  and  effective  use  of  the 
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measurement  program  is  a  good  blend  of  effective  planning  and  open,  honest 
communication. 

3.2.1  Management  Commitment 

A  critical  aspect  of  the  infrastructure  of  a  measurement  program  is  the  level  of  support 
that  is  provided  by  management.  Management’s  commitment  to  the  objectives  of  the 
measurement  program  must  be  both  visible  and  well  communicated.  Without  this  visible 
and  continuing  support,  the  infrastructure  of  the  program  will  not  be  properly  prepared 
nor  sustained,  and  the  likelihood  of  success  of  the  program  dwindles.  In  addition, 
management  needs  to  provide  ongoing  feedback  to  the  practitioners  in  the  measurement 
program.  The  practitioners  need  to  know  that  the  data  collected,  the  analysis  performed, 
and  the  recommendations  provided  were  useful  and  led  to  beneficial  insight  that  influenced 
management’s  decisions.  This  communication  has  a  valuable  intrinsic  benefit. 

3.2.2  Measurement  Plan 

The  measurement  plan  is  the  documented  result  of  management’s  objectives,  and  the 
process  activities  for  identification  of  issues,  selecting  appropriate  measures,  and 
specifying  those  measures.  The  plan  should  be  used  to  guide  the  implementation  and 
execution  of  the  measurement  program.  Also,  it  should  be  a  living  document  that  is 
updated  whenever  issues  and  associated  measures  change.  The  following  is  a  list  of  typical 
contents  of  a  measurement  plan. 

•  Issues  and  Measures  selected 

•  Identification  and  definition  of  data  elements  to  be  collected  (as  negotiated) 

•  Data  Collection  Details 

=>  Data  sources 
=>  Level  of  measurement 
=>  Units  of  measure 
=>  Method  of  collection 
=>  Frequency 

•  Data  Delivery  Details 

=>  Aggregation  structure 
=>  Frequency  of  delivery 
=>  Method/format  of  delivery 

•  Communication  process  and  POC  interfaces 

•  Analysis  and  reporting  details  and  guidance 

=>  Type  of  general  analysis 

=>  Frequency  of  analysis  and  reporting 

=>  Criteria  for  additional  or  specialized  analysis 
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3.2.3  Resources 

As  with  any  set  of  tasks,  the  measurement  program  requires  careful  planning, 
implementation,  and  control.  All  tasks  should  be  identified,  scheduled,  and  have  effort  and 
resource  estimations  performed.  For  new  measurement  programs,  this  planning  includes 
the  tasks  associated  with  the  initial  implementation  of  the  measurement  program.  Careful 
consideration  needs  to  be  given  to  the  scope  of  the  measurement  tasks,  the  support 
required  across  the  entire  life  cycle,  and  the  assignment  of  the  measurement 
responsibilities  within  the  organization. 

3.2.4  Training 

The  appropriate  level  of  training  must  be  planned  for  and  provided  to  all  stakeholders  of 
the  measurement  process.  All  stakeholders  need  to  understand  the  purpose  of  the 
measurement  program  and  measures  selected.  In  addition,  the  following  is  a  list  of  the 
typical  measurement  roles  and  applicable  tasks  for  which  training  may  be  necessary: 


ROLES 

Data  Measurement  Decision  Process 


TASKS 

Collectors 

Analysts 

Makers 

Owners 

Issue  Identification 

X 

X 

X 

Selection  of  Measures 

X 

X 

X 

Specification  of  Measures 

X 

Measurement  Planning 

X 

X 

Data  Collection  Methods 

X 

X 

Data  Collection/Storage  Tools 

X 

X 

Data  Validation  &  Verification 

X 

X 

X 

Data  Preparation  (Normalization, 
Aggregation,  etc.) 

X 

Analysis/Statistical  Methods/T  ools 

X 

Measurement  Interpretation 

X 

X 

X 

Reporting  Techniques/Tools 

X 

Decision  Support  Techniques/Tools 
Usage  of  Results 

X 

X 

X 

Feedback  to  Practitioners 

X 

3.2.5  Tools 

Automate,  automate,  automate!  This  is  a  common  mantra  of  many  experienced 
measurement  practitioners.  Very  few  successful  measurement  programs  exist  that  do  not 
have  tools  that  automatically  support  some  portion  of  the  data  collection,  analysis,  and 
reporting.  These  tools  can  often  reduce  the  effort  needed  to  support  a  measurement 
program,  as  well  as  the  potential  for  biased  data.  Depending  on  the  tools  chosen,  support 
for  data  preparation,  such  as  normalization,  or  analysis  capabilities,  such  as  linear 
regression,  could  also  be  available.  Some  practitioners  even  employ  decision  support 
tools  to  aid  the  evaluation  of  alternatives  in  order  to  provide  management  with  the  best 
recommendation  or  corrective  action. 
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There  is  a  risk  of  automating  too  much  of  the  program,  though.  Do  not  automate  for  the 
sake  of  automation!  Remember  that  tools  have  costs  for  procurement,  support,  training, 
and  labor.  Ask  the  following  two  questions: 

•  “Will  incorporating  and  maintaining  this  tool  require  more  effort  than  doing  the  work 
manually?” 

•  “Will  the  licensing  and  maintenance  costs  of  the  tool  outweigh  any  savings  in  labor?” 

If  the  answer  to  either  of  these  is  “yes”,  then  the  process  should  remain  a  manual  one, 
unless  there  it  is  expected  that  there  will  be  a  significant  improvement  in  accuracy  or 
usability  of  the  data  and  results.  In  general,  use  of  tools  will  reduce  the  labor  and  improve 
the  effectiveness  of  the  measure  program,  leading  to  a  higher  probability  of  success. 

3.2.6  Measurement  Data  Repository 

Data  storage  and  retrieval  must  be  planned  and  adequately  supported.  The  storage  and 
retrieval  mechanism  may  be  either  an  automated  database  management  system  or  via  a 
manual  repository  of  records.  In  either  case,  thought  should  be  given  to  the  organization 
of  the  repository  and  to  the  interaction  of  the  data  repository  with  any  other  data 
management  tools.  Without  proper  planning  and  ongoing  support,  the  data  can  become 
unreliable,  difficult  to  obtain  in  a  timely  manner,  and  more  labor  intensive  to  use. 


17 


4.  Purposes  of  Measurement 

Each  organization’s  processes  and  products  are  unique;  therefore,  any  measurement 
program  implemented  for  a  particular  process  or  product  must  be  tailored  to  that 
organization.  However,  some  general  purposes  and  guidelines  for  measurement 
performance  and  usage  are  shared  by  experienced  measurement  practitioners  pertaining  to 
typical  purposes  and  uses  of  measurement. 

4.1  Defining  and  Communicating  the  Purpose  of  Measures 

The  purpose  of  a  measure  is  to  provide  meaningful  information  regarding  the  quality, 
adequacy,  and/or  progress  of  process,  project,  and/or  products.  Measurement  does  not  in 
itself  result  in  improvement.  However,  measures  offer  the  insight  needed  for  planning, 
controlling,  managing,  and  improving: 

•  Product  technical  adequacy  and  performance, 

•  Schedule  and  progress, 

•  Resources  and  cost, 

•  Growth  and  stability, 

•  Product  quality, 

•  Life  cycle  process  performance, 

•  Technology  effectiveness,  and/or 

•  Customer  satisfaction. 

The  objective  of  measurement  is  to  obtain  insight  into  issues  that  impact  project  cost, 
schedule,  and  technical  (performance,  functionality,  and  quality)  objectives  in  order  to 
enable  the  program  decision  makers  to  make  informed  decisions.  Applied  systematically 
throughout  the  project,  measurement  helps  to: 

•  Identify  specific  problems 

•  Assess  the  impacts  of  these  problems  with  respect  to  all  program  control  factors  (cost, 
schedule,  quality,  functionality,  and  performance) 

•  Determine  feasible  alternative  solutions 

•  Perform  tradeoff  analysis  and  select  the  optimal  approach  (for  all  tasks  and  corrective 
actions) 

•  Provide  accountability  of  decisions 

•  Communicate  progress,  performance,  and  problems  in  a  more  precise,  standard,  objective 
manner  throughout  the  organization 

When  the  purpose  of  a  measure  is  clearly  defined,  the  chances  of  achieving  that  purpose 
are  greatly  improved.  In  selecting  measures  of  interest,  it  is  valuable  to  consider  the 
effectiveness  of  the  measure  in  achieving  multiple  purposes,  including: 

•  Assessing  progress  in  meeting  performance  objectives 

•  Identifying  opportunities  to  improve  process  and  product  and  evaluating  improvement 
results 
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•  Developing  projections  and  plans  with  greater  confidence 

•  Providing  feedback  on  status  and  progress 

•  Enabling  quantitative  process  management 

Regardless  of  the  defined  purpose  of  the  measures,  the  implementation  and  execution  of 
the  measurement  program  must  be  done  in  a  manner  that  builds  trust  and  support 
throughout  the  organization.  Doubts  can  arise  concerning  the  legitimacy  of  the  measures 
or  the  value  added  to  the  process  or  product  that  is  being  measured  and  analyzed.  There  is 
often  resistance  to  the  “scrutiny”  of  measurement,  even  when  the  true  intention  is  a 
beneficial  one  such  as  identifying  ways  to  improve  processes  or  products.  Clearly  defining 
the  measures’  purpose  will  help  reduce  the  resistance  and  the  potential  for  human  bias 
(often  unintentional)  from  creeping  into  the  data,  and  therefore,  adding  credibility  to  the 
“objective  insight”  purpose  of  measurement.  Involving  stakeholders  in  the  process  from 
the  beginning  will  also  improve  tmst  and  buy-in.  Achieving  acceptance  for  a  measurement 
program  in  an  organization  requires  these  plans  and  benefits  be  propagated  to  both  those 
individuals  who  are  involved  in  the  measurement  activity,  as  well  as  to  those  who  are  the 
recipients  of  measurement  analysis  results. 

4.1.1  Characterization:  Gain  Understanding  of  Products  and  Processes 

4. 1.1.1  Measuring  Technical  Performance 

Technical  Performance  Measures  (TPMs)  are  critical  technical  parameters  that  a  project 
monitors  to  ensure  that  the  technical  objectives  of  a  product/project  will  be  realized. 
Typically,  TPMs  have  planned  values  at  defined  time  increments,  against  which  the 
estimated  and  actual  values  are  plotted.  Collection  of  TPMs  allows  trend  detection  and 
correction,  and  supports  risk  identification  and  assessment,  thereby  providing  feedback 
information  to  identify  potential  performance  problems  prior  to  incurring  significant  cost 
or  schedule  impacts.  In  the  engineering  of  a  system,  TPMs  are  a  very  common  form  of 
product  measurement  that  also  provides  insight  into  project  accomplishment  towards  the 
technical  goals. 

TPMs  are  selected  by  identifying  the  elements  of  the  system  for  which  performance  is 
critical.  Review  of  the  systems  engineering  documentation  and  requirements 
specifications  help  identify  these  elements  and  the  necessary  planned  values.  However,  the 
number  of  TPMs  selected  at  the  system  level  must  be  kept  small  to  be  cost-effective,  since 
each  system  level  TPM  may  be  supported  by  several  lower  level,  more  detailed 
parameters.  Thus,  only  the  most  critical  parameters  should  be  selected  and  the  following 
criteria  should  be  considered  in  the  selection  process: 

•  The  parameter  is  one  of  the  best  indicators  of  the  total  system  performance 

•  The  parameter  is  directly  measurable  and  easy  to  analyze  and  interpret 

•  The  parameter  can  be  defined  with  relatively  meaningful  tolerance  bands 

TPMs  can  be  related  to  any  functional  aspect  of  the  system;  software,  hardware, 
operations,  or  logistics.  (See  the  DSMC  SEMG  reference  for  more  TPM  information.) 
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An  example  of  a  TPM  is  the  weight  of  the  end-product,  such  as  an  aircraft,  where  the 
product  must  be  delivered  weighing  less  than  a  certain  amount.  Throughout  the  project, 
as  components  are  selected  and  integrated,  the  overall  weight  is  calculated  and  monitored. 


4.1. 1.2  Measuring  Process  Performance 

Process  measurement  is  critical  in  the  determination  of  process  efficiency  and 
effectiveness.  With  most  organizations  trying  to  do  more  with  less  resources,  there  has 
been  considerable  focus  in  recent  years  on  process  improvement.  Most  of  the  process 
improvement  frameworks,  such  as  the  various  capability  models,  require  some  level  of 
process  measurement  at  each  level  above  the  model’s  initial  level. 

Process  performance  measurement  starts  with  identifying  the  measurable  attributes  of  the 
process.  These  should  then  be  assessed  against  organizational  objectives  to  select  the 
attributes  that  will  be  measured.  Periodic  measurement  of  these  attributes  are  the  basis  of 
characterizing  the  organization’s  performance  of  that  process.  The  data  can  be  tracked 
against  the  business  objectives  to  evaluate  the  performance  and  used  to  identify  those 
attributes  of  the  process  that  are  the  highest  priority  to  improve.  (See  SEI  MPM  reference 
for  more  information.) 

4.1.2  Improvement:  Identifying  and  Evaluating  Process  and  Product 
Improvement  Opportunities 

Measures  are  analyzed  and  combined  to  form  indicators  that  provide  the  insight  needed  to 
identify  opportunities  for  improvement.  Opportunities  for  improvement  are  identified  by 
analyzing  actual  measured  process,  product,  or  project  attributes  against  target  values  and 
business  objectives.  Where  variances  exist  are  potential  areas  for  improvement 
opportunities.  These  are  then  evaluated  and  prioritized  based  on  the  severity  of  the 
variance  and  the  importance  of  the  objectives.  When  improvements  are  made,  measures 
are  essential  to  discerning  if  the  improvement  activity  has  a  favorable  outcome.  Although 
anecdotal  information  can  indicate  that  a  process  or  product  has  improved,  only  measures 
can  quantify  the  improvement.  Of  course,  the  measurement  information  must  be  recorded 
in  the  same  manner  for  both  the  initial  and  end  states.  As  an  example,  a  measure  such  as 
specification  defects  can  be  monitored  in  an  effort  to  improve  the  specification 
development  process  and  provide  evidence  that  the  improvement  action  was  successful. 

4. 1.2.1  Enabling  Quantitative  Process  Management  (QPM) 

The  purpose  of  QPM  is  to  control  the  process  performance  of  a  project  quantitatively. 
QPM,  a  key  element  of  process  maturity  and  assessment  models,  such  as  the  INCOSE 
Systems  Engineering  Capability  Assessment  Model  (SECAM),  involves  establishing  goals 
for  performance  of  processes,  collecting  and  analyzing  the  measures  of  process 
performance,  and  making  adjustments  to  maintain  process  performance  within  acceptable 
limits.  This  goal  is  more  extensive  than  simple  process  improvement;  it  requires  every 
player  in  an  organization  to  be  involved  in  not  only  the  collection  and  analysis  of 
measures,  but  also  in  the  use  of  measures  to  aid  identification  and  monitoring 
improvement  opportunities  in  every  element  of  the  performance  of  business  processes. 
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4.1.3  Prediction:  Facilitating  Projections  and  Planning 

Projections  and  planning  are  improved  by  the  availability  of  historical  data.  The  data  is 
used  to  formulate  statistical  and  causal  models  for  predictions.  New  projects  can  be 
budgeted,  scheduled,  and  planned  much  more  effectively  if  measures  which  provide 
historical  data  on  previous  similar  projects/products  are  used.  Organizations  begin  by 
compiling  a  database  of  information.  As  more  data  is  collected,  better  baselines  and 
control  limits  are  established.  Periodically,  estimation  models  should  be  calibrated  against 
this  historical  data. 

4.1.4  Evaluation:  Providing  Feedback  and  Status 

Measures  can  provide  valuable  feedback  to  the  team  or  customer.  A  measure  such  as 
customer  satisfaction ,  typically  measured  using  a  survey,  provides  feedback  on  the  overall 
satisfaction  with  a  product  or  service.  A  product  penetration  measure  provides  feedback 
on  how  well  the  organization  has  done  in  penetrating  a  given  market.  Team  effectiveness 
measures  based  on  surveys  provide  feedback  to  the  team  on  how  effective  the  individual 
members  perceive  the  team  to  be. 

Measures  are  an  effective  status  reporting  tool  as  well,  particularly  when  presented  in 
graphical  form.  An  example  of  such  a  measure  might  be  on-time  deliveries  or 
requirements  verification  completeness.  The  purpose  is  to  provide  the  team  with 
quantified  information  related  to  process,  project,  and  or  product,  particularly  in  terms  of 
status,  progress,  completion,  and  potential  problems.  Keep  in  mind,  however,  that  during 
the  early  stages  of  a  new  measurement  program,  the  measures  themselves  can  be  very 
volatile.  In  this  event,  careful  consideration  and  qualification  of  measurement  results  may 
be  warranted. 

Measurement-based  feedback  improves  the  effectiveness  of  the  project  team  in  such  ways 
as: 

•  Analyzing  trends  that  help  focus  on  problem  areas  at  the  earliest  point  in  time, 

•  Providing  early  insight  into  error  prone  products  that  can  then  be  corrected  at  the  lowest 
cost, 

•  Avoiding  or  minimizing  cost  overruns  and  schedule  slips  by  detecting  them  early  enough  in 
the  project  to  implement  corrective  actions, 

•  Identifying  complexities  in  the  design  or  technical  performance  progress  to  enable  a  focus 
on  risk  areas,  and 

•  Making  adjustments  to  resources  based  on  discrepancies  between  planned  and  actual 
progress. 
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5.  Application  Guidance  and  Lessons  Learned 

For  those  embarking  for  the  first  time  into  the  world  of  measurement,  and  perhaps 
beginning  to  plan  a  measurement  program,  there  are  some  factors  about  measurement  and 
lessons  learned  that  are  helpful  to  know.  A  description  of  the  right  and  wrong  ways  to  use 
measurement,  some  helpful  hints  about  creating  or  maintaining  a  good  measurement 
program,  and  some  considerations  of  the  influence  that  human  factors  can  have  on 
measurement  programs  are  provided  here  for  your  assistance. 

5.1  Contrasting  Correct  and  Incorrect  Uses  of  Measurement 

Measures  that  are  used  correctly  will  promote  understanding  and  motivate  action.  But 
what  happens  if  a  measure  is  used  incorrectly?  What  is  the  right  or  wrong  way  to  use 
measures?  This  section  answers  those  questions  by  listing  the  effects  of  using  measures 
both  in  a  constructive  way  and  in  a  destructive  way. 

A  measure  used  correctly: 

•  Is  accepted  as  having  value  to  the  customer  or  as  an  attribute  essential  to  customer 
satisfaction.  The  measurement  customers  can  be  people  internal  or  external  to  the 
organization,  the  stakeholders  of  the  measurement  program,  or  even  the  users  of  the  measure. 

•  Tells  how  well  organizational  goals  and  objectives  are  being  met  through  processes  and  tasks. 

•  Stands  the  test  of  time.  A  measure  that  is  used  continually  and  is  an  accepted  part  of  an 
organization’s  processes  is  most  likely  one  that  is  being  beneficial  to  that  organization. 

•  Is  the  basis  of  indicators  that  provide  insight  that  drives  appropriate  action. 

However,  measures  are  not  always  used  appropriately.  How  do  you  know  if  a  measure  is 
being  abused?  You  can  recognize  an  incorrectly  used  measure  if  it: 

•  Is  a  chart  or  display  tool  (these  may  be  used  to  present  the  results  of  a  measure,  but  the 
measure  does  not  consist  solely  of  them). 

•  Is  interpreted  as  team  or  personnel  control  tools.  When  the  measure  is  purposefully  used  in 
this  way,  a  natural  result  will  be  people  feeling  distrustful,  and  “gaming”  of  the  system  will 
occur. 

•  Lasts  forever.  Not  all  measures  are  appropriate  to  all  phases  of  the  lifecycle.  There  is  no 
“perfect”  measure;  measures  need  to  adapt  to  products  and  processes  as  they  evolve. 

•  Is  a  scheduling  tool.  However,  schedules  (i.e.,  plans)  form  the  basis  of  some  good  measures. 

•  Is  a  count  of  activity.  This  data  becomes  useful  only  when  transformed  into  knowledge  that 
can  lead  to  action. 

•  Is  used  as  an  absolute.  A  single  measure  by  itself  is  rarely  an  absolute  indicator  of  anything. 

•  Is  used  to  compare  people,  departments,  or  other  human  factors. 
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5.2  Tips  and  Rules  of  Thumb 

Although  measurement  cannot  guarantee  program  success,  it  will  provide  insight  to  make 
better  decisions.  The  objective  is  to  obtain  the  best  insight  possible.  To  facilitate  this,  a 
set  of  lessons  learned  from  successful  measurement  practitioners  has  been  pulled  together. 
This  set  of  tips  and  rules  of  thumb  for  making  measurement  a  more  effective  project 
management  tool  and  increasing  the  measurement  program’s  probability  of  success  are  as 
follows: 

•  Project  issues  (risks,  concerns,  constraints,  and  objectives)  should  drive  the  measures  and 
indicators  selected.  Never  perform  measurement  for  measurement’s  sake  or  for  purely 
“political”  reasons.  Constrain  the  measures  to  those  measuring  process,  product,  or  project 
attributes  that  require  additional  insight. 

•  Make  sure  that  any  standard  (core)  set  of  measures  is  kept  small  (try  not  to  exceed  6).  This 
will  allow  for  comparison  across  the  organization  and  across  projects  without  significantly 
impacting  the  use  of  measures  for  project  specific  insight.  Additional  measures  should  be 
based  on  project/product  specific  goals,  objectives,  issues,  and  risks. 

•  Assign  a  measurement  process  “owner.” 

•  Measures  and  their  analysis  should  be  traceable  to  issues,  decisions  and  corrective  actions. 
This  allows  better  evaluation  of  the  usefulness  of  the  measures  and  provides  better  project 
accountability. 

•  Re-evaluate  your  measurement  program  at  regular  intervals.  Is  it  working?  Is  adjustment 
needed  to  the  process  or  measures  used? 

•  You  cannot  measure  what  is  not  defined.  For  process  measures,  ensure  you  have  a  process 
defined  first. 

•  Historically,  people  resist  the  idea  of  being  measured.  Find  a  way  to  use  measurement  in  a 
positive,  team  unifying  role. 

•  Measurement  should  be  an  integral  part  of  the  program  management  process  throughout  the 
entire  life  cycle  and  used  as  part  of  the  basis  of  decisions. 

•  The  measurement  process  should  be  a  planned  and  natural  part  of  the  technical  processes.  This 
will  minimize  the  amount  of  effort  needed  to  collect  data.  Where  possible  select  measures  that 
require  data  that  is  naturally  available  in  the  development  process.  A  data  collection  method 
that  can  be  tailored  to  become  part  of  the  routine  work  that  must  occur  anyway  has  the  best 
chance  of  not  being  labeled  as  an  additional  burden. 

•  Try  to  automate  data  collection  and  reporting  as  much  as  possible.  This  removes  potential 
bias  in  the  data  and  provides  data  in  a  regular,  timely  manner. 

•  Measurement  results  should  be  interpreted  in  the  context  of  other  sources  of  information. 
When  measurement  results  are  combined  with  other  program  data,  better  insight  can  be  gained 
of  actual  problem  existence  and  the  root  cause  of  the  problems.  This  insight  is  needed  to  make 
effective  decisions.  The  types  of  data  that  are  combined  are  dependent  on  the  puipose  of  the 
measurement. 

•  Estimation  models  should  be  calibrated  with  actual  historic  data. 

When  analyzing  measurement  data,  keep  these  tips  in  mind: 
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•  A  tool  introduced  to  “observe”  processes  and/or  products  will,  by  its  very  nature,  influence  the 
“output.”  Be  aware  of  the  influence  that  introducing  a  measurement  program  is  having  to 
avoid  misdiagnosing  an  improvement  as  being  the  result  of  process  or  product  changes. 

•  There  may  be  multiple  factors  affecting  the  results,  confounding  the  analysis  and  the  decision 
maker. 

•  Correlation  between  two  variables  does  not  imply  that  changing  one  causes  a  change  in  the 
other! 

•  Projecting  a  trend  based  on  history  is  likely  to  provide  incorrect  interpretations  if  you  do  not 
understand  all  of  the  underlying  factors.  For  example,  say  you  identify  a  trend  of  having  a  low 
number  of  defects  found  per  widget.  You  might  be  tempted  to  expect  to  continue  to  measure 
low  numbers  of  defects.  But  if  you  are  not  in  a  test  mode,  then  it  would  be  expected  that  low 
numbers  of  defects  would  be  found  until  a  test  mode  is  entered.  A  sudden  jump  in  the  number 
of  defects  found  per  widget  would  be  unexpected  unless  the  underlying  factor  of  current  mode 
of  operation  is  taken  into  account  when  analyzing  historical  data. 

•  Scaling  factors  and  weights  can  distort  or  hide  information. 

As  with  any  good  organizational  initiative,  a  successful  measurement  program  requires 
preplanning.  This  planning  will  improve  management’s  ability  to  obtain  and  comprehend 
the  information  yielded  by  the  measurement  results  and  assists  making  appropriate 
improvement  changes.  Encouraging  management  to  define  their  goals  and  expectations 
for  achievement  will  also  increase  their  participation  in  the  preplanning  phases.  As  a 
minimum,  the  planning  effort  should  include  consideration  and  definition  of  the  following: 

•  Project/process/product  issues.  The  issues  identified  drive  the  aspects  of  the  project,  process, 
or  product  which  are  targeted  to  be  measured  and  improved.  Some  examples  include  any 
organizational  goals  and  objectives,  process  improvement  objectives  and  evaluation 
criteria/guidelines,  risks,  concerns,  and  programmatic  or  technical  constraints. 

•  Measures  and  their  specifications.  As  a  minimum,  specification  of  the  measures  includes  its 
definition,  data  collection  required,  and  interpretation  notes. 

•  Specification  of  data  flow.  This  could  include  data  sources,  frequency  of  data  collection,  the 
method  for  collection,  and,  if  required,  the  delivery  method  of  the  measurement  results. 

•  Measurement  aggregation  structures  (i.e.,  what  level  of  summarization  is  needed  to  provide  the 
appropriate  insight  for  each  level  of  decision). 

•  Frequency  and  type  of  analysis  and  reporting 

•  Lines  of  communication  and  interfaces  (defining  discrete  channels  for  open  and  honest  two- 
way  communication). 

•  The  roles  and  responsibilities  of  all  involved  parties.  Some  typical  roles  that  are  defined  for 
measurement  programs  are  the  measurement  analyst,  the  collector  of  the  measures,  and  the 
customer  of  the  measurement  results,  which  can  be  the  program  management  team,  the  end- 
product  customer,  and  other  decision  makers. 

•  The  awareness  and  cooperation  of  the  organizational  cultural  changes  that  will  be  made. 
Possible  ways  to  help  people  become  aware  of  the  changes  that  will  occur  are  to  support 
training,  mentoring,  open  discussions,  policies  that  minimize  misuse  of  measurement,  and  the 
communication  of  the  expected  reactions  to  the  changes. 
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5.3  Human  Factors 

The  cultural  and  organizational  changes  and  implications  which  are  not  adequately 
anticipated  often  become  the  major  obstacle  in  establishing  a  successful  measurement 
program.  For  example,  performance  measures  of  a  process  will  often  provide  information 
and  insight  that  ultimately  leads  to  a  more  effective  and  efficient  process.  This  enhanced 
process  may  require  less  effort  and  reduced  staff  to  perform  the  same  function. 
Management  should  address  the  need  to  belay  employees’  anxiety  of  job  loss,  while  still 
promoting  the  trustful  environment  of  process  introspection  that  is  required  for 
maintaining  a  viable  measurement  program.  Without  proper  planning  for  and 
communications  regarding  these  factors,  data  will  be  biased  and  the  measurement  program 
may  be  rendered  ineffective. 
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6.  List  of  References 

Measurement  as  a  topic  is  very  broad  in  scope.  There  is  a  wealth  of  information  about 
measurement  in  many  different  industries  and  organizational  types.  Since  organizations 
may  need  information  on  more  specialized  topics  in  measurement  after  understanding  the 
basics,  this  section  of  the  Primer  was  created  to  address  that  need.  The  following  list  of 
resources  are  organized  according  to  specialized  topics  in  measurement.  Each  sub-topic 
includes  a  short  synopsis  of  the  information  available  in  it.  Many  of  the  concepts 
introduced  in  other  sections  of  the  Primer  overlap  with  concepts  discussed  in  the  list  of 
sources  below.  However,  because  only  generalities  are  described  in  this  document,  no 
source  is  quoted  verbatim  and  no  references  to  other  sources  are  made  in  this  Primer 
outside  of  this  section.  This  list  of  sub-topics  is  not  exhaustive;  if  good  references  exist 
that  discuss  a  particular  sub-topic  that  is  not  listed  here,  please  follow  the  feedback 
instructions  in  section  8  to  pass  along  that  information. 

6.1  Measurement  References 

For  more  detailed  information  on  the  topic  of  measurement,  consult  the  following 
references.  These  sources  provide  a  more  comprehensive  introduction  to  measurement. 

•  Metrics  Guidebook  for  Integrated  Systems  and  Product  Development ,  INCOSE,  July 

1995. 

•  Practical  Software  Measurement:  A  Guide  to  Objective  Program  Insight  [PSM],  Joint 
Logistics  Commanders,  Joint  Group  on  Systems  Engineering,  Version  2.1,  March 

1996.  (The  measurement  process  provided  in  this  reference  is  an  easy  to  follow 
process  that  can  be  applied  to  any  type  of  measurement,  not  just  software.  However, 
the  measures  discussed  are  software  project  measures.) 

•  Metrics  Starter  Kit ,  Air  Force  Software  Technology  Support  Center,  February  1994. 

•  Metrics  Starter  Kit ,  AMI’s  Center  for  System  and  Software  Engineering.  (Southbank 
University,  103  Borough  Rd.,  London,  SE10AA,  UK.) 

6.2  Selection  or  Creation  of  a  Metric 

If  a  specific  type  of  measure  needs  to  be  created,  for  example  a  customer  satisfaction 
measure,  then  it  is  helpful  either  to  have  guidance  on  how  to  create  it  or  to  use  one  that 
already  exists.  This  section  includes  sources  that  will  guide  you  on  how  to  create  a 
specific  type  of  measure. 

•  Moody,  J.,  Chapman,  W.,  Van  Voorhees,  F.  and  Bahill,  A.,  Metrics  and  Case  Studies 
for  Evaluating  Engineering  Designs ,  Prentice-Hall,  1997.  (This  book  provides 
detailed  information,  including  case  studies,  on  the  use  of  design  difficulty  and 
resource  measures.) 

•  Defense  Systems  Management  College,  Systems  Engineering  Management  Guide , 
DSMC  Press,  Fourth  Edition  (Draft)  [DSMC  SEMG],  1996  (This  guide  has  good 
information  on  the  selection  and  use  of  Technical  Performance  Measures.) 
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•  Kan,  S.,  Metrics  and  Models  in  Software  Quality  Engineering,  Addison- Wesley 
Publishing  Company,  1995.  (This  book  provides  information  on  the  selection  and  use 
of  software  quality  measures.) 

•  Putnam,  L.  and  Myers,  W.,  Measures  for  Excellence ,  Prentice-Hall,  1992.  (This  book 
provides  information  on  the  selection  and  use  of  project  performance  and  software 
quality  measures.  Much  of  the  information  applies  equally  to  the  systems 
environment.) 

•  Cmickshank,  R.,  Gaffney,  J.  and  Weling,  R.,  “Measurement  at  the  systems  level: 
where  we  are  and  where  we  ought  to  be,  ”  Proceedings  of  the  4th  National  Council  on 
Systems  Engineering,  August  1994.  (This  paper  describes  how  the  measurement 
process  includes  the  selection  of  measures;  the  collection  of  data;  and  the  integration, 
analysis,  and  reporting  of  quantitative  information  derived  from  the  data.) 

•  Gause,  D.D.  and  Weinberg,  G.M.,  Exploring  Requirements:  Quality  Before  Design, 
Dorset  House  Publishing,  1989.  (This  book  includes  discussion  on  ambiguity 
measures  and  customer  satisfaction  measures.) 

•  Hoffman,  K.C.,  “Management  of  enterprise-wide  systems  integration  programs,” 
Proceedings  of  the  Second  International  Conference  on  Systems  Integration,  (Cat.  No. 
92TH0444-0),  P.  4-13,  IEEE  Comput.  Soc.  Press,  1992.  (This  paper  describes 
measures  in  system  integration  programs.) 

•  Kasser,  J.  and  Schermerhom,  R.,  “Determining  metrics  for  systems  engineering,” 
Proceedings  of  the  4th  National  Council  on  Systems  Engineering,  August  1994.  (This 
paper  suggests  some  product  and  progress  measures.) 

•  Martin,  J.N.,  “On  the  diversity  of  process  metrics  users:  identification  of  metrics  types 
and  other  attributes,”  Proceedings  of  the  5th  National  Council  on  Systems 
Engineering,  July  1995.  (This  paper  identifies  specific  types  of  measures.) 

•  Thomas,  H.D.,  “On  management  and  metrics,”  Proceedings  of  the  3rd  National 
Council  on  Systems  Engineering,  July  1993.  (This  describes  a  Customer  Satisfaction 
Metric.) 

•  Patterson,  Marvin,  and  Sam  Lightman,  Accelerating  Innovation  -  Improving  the 
Process  of  Product  Development,  Van  Nostrand  Reinhold,  New  York,  1993.  (Chapter 
3  -  Designing  Metrics) 

6.3  Starting  a  Measurement  Program 

Measurement  is  not  useful  without  a  defined  process  which  outlines  how  and  when  to 

collect  the  data,  how  and  when  to  analyze  the  data,  etc.  These  sources  give  guidelines  on 

how  to  create  a  measurement  program  that  will  support  measurement. 

•  Grady,  Robert  B.,  Practical  Software  Metrics  for  Project  Management  and  Process 
Improvement,  Prentice-Hall,  1992. 

•  Hetzel,  Bill,  Making  Software  Measurement  Work:  Building  an  Effective 
Measurement  Program,  John  Wiley  &  Sons,  Inc.,  1993. 

•  Grady,  Robert  B.,  Caswell,  D.L.,  Software  Metrics:  Establishing  a  Company-wide 
Program,  Prentice  Hah,  1987. 
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•  Fisher,  G.H.,  “Startup  guide  for  systems  engineering  metrics,  ”  Proceedings  of  the  3rd 
National  Council  on  Systems  Engineering ,  July  1993. 

•  Rhodes,  D.  and  Mastranadi,  A.,  “Getting  started  with  a  measurement  program,  ” 
Proceedings  of  the  2nd  National  Council  on  Systems  Engineering ,  July  1992. 

•  Donnell,  A.  and  Dellinger,  M.,  Analyzing  Business  Process  Data:  The  Looking  Glass, 
AT&T  Quality  Steering  Committee,  AT&T,  1990. 

6.4  Improving  an  Existing  Measure 

How  effective  is  a  measure?  Is  it  measuring  what  it  is  intended  to  measure?  These 

sources  explain  how  to  examine  a  measure  and  improve  it. 

•  Ashbumer,  E.J.,  “The  theory  of  testing  applied  to  the  development  of  a  system 
engineering  metric,”  Proceedings  of  the  4th  National  Council  on  Systems 
Engineering,  August  1994. 

•  Martin,  J.N.  and  Miller,  W.D.,  “Re-engineering  of  the  metrics  collection  process,” 
Proceedings  of  the  5th  National  Council  on  Systems  Engineering,  July  1995. 

•  Miller,  W.  D.,  “Systems  engineering  metrics  ++,”  Proceedings  of  the  4th  National 
Council  on  Systems  Engineering,  August  1994. 

•  Moller,  K.H.  and  D.J.  Paulish,  Software  Metrics:  A  Practitioner’ s  Guide  to  Improved 
Product  Development,  IEEE,  1993. 

6.5  Introduction  to  Statistics,  SPC,  and  Process  Improvement 

The  following  sources  contain  information  on  statistical  concepts  and  their  uses  in 

management  and  business  for  process  improvement  and  control. 

•  Practical  Software  Measure:  Measurement  for  Process  Management  and 
Improvement  [SEI  MPM],  Software  Engineering  Institute,  Guidebook  CMU/SEI-97- 
HB-003,  April  1997. 

•  Wheeler,  Donald  J.,  The  Key  to  Managing  Chaos  and  Understanding  SPC,  SPC  Press 
Inc.,  1996. 

•  Pitt,  H.,  SPC  for  the  Rest  of  Us:  A  Statistical  Process  Control,  ASQC  Press,  1996. 

•  Keats  and  Montgomery,  Statistical  Applications  in  Process  Control,  Mercel  Dekker, 
Inc.,  1996. 

•  Dovich,  R.,  Quality  Engineering  Statistics,  ASQC  Quality  Press,  1992. 

•  Mandel  and  Laessig,  Statistics  for  Management,  Douglas  Publishing  Co.,  1996. 
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7.  Example  Measures 

In  Section  2,  three  different  types  of  measures  were  defined;  Process,  Product,  and 
Progress.  This  section  will  illustrate  a  measure  from  each  area  and  how  they  are  used  in 
the  systems  development  process.  The  measures  chosen  for  this  section  were  selected  as 
being  representative  of  each  of  the  types  of  measure.  They  are  not  necessarily  advocated 
as  the  measure  one  should  use  for  all  organizations.  As  emphasized  earlier,  the  measures 
selected  should  be  based  upon  specified  goals  and  objectives.  Remember  that  in  an  actual 
application  of  these  or  any  other  measures,  all  applicable  measures  and  program 
information  that  is  available  should  be  used  to  support  making  decisions.  Use  extreme 
caution  if  you  decide  on  a  course  of  action  based  upon  a  single  measure  in  isolation! 

The  format  that  will  be  followed  for  each  example  will  be  the  same  as  that  employed  in 
another  Measurement  Working  Group  Product  called  the  Metrics  Information  Systems 
Tool  (MIST),  namely:  the  potential  usage;  the  source  of  the  measure;  the  appropriate 
user(s);  how  the  data  is  collected  and  at  what  interval;  the  basic  algorithm  used  in 
calculating  the  measure  or  indicator;  and  the  interpretation  of  the  results. 

7.1  Example  Process  Measure:  Review  Rate 

This  measure  is  obtained  from  the  Development  Team  and  can  be  used  by  the 
management,  the  development  team,  and  any  individuals  concerned  with  the  quality  of  the 
review  process.  The  number  is  usually  calculated  on  a  project  basis  but  compared  against 
the  results  from  similar  projects.  This  comparison  might  involve  using  statistical  control 
charts  to  insure  that  adequate  time  is  being  spent  in  the  review  process. 

This  measure  is  obtained  by  dividing  some  measure  of  size  for  the  project  by  the 
cumulative  amount  of  time  spent  in  preparing  for  the  review.  Size  can  be  determined  by 
such  factors  as:  number  of  requirements,  number  of  documentation  pages,  number  of  test 
items  covered,  or  number  of  lines-of-code  covered  at  the  review.  The  important  issue  is 
to  agree  on  one  measure  of  size  and  then  use  it  for  all  the  projects  so  that  valid 
comparisons  can  be  drawn.  This  rate  can  be  calculated  for  designated  reviews  during  the 
development  process  and  then  compared  against  other  projects’  review  rates  from  their 
equivalent  stages  in  development.  If  a  given  project  exceeds  the  statistical  control  limits, 
the  development  team  needs  to  ask  why.  This  rate  can  also  be  correlated  with  number  of 
errors  found. 

Figure  2  is  an  example  of  a  plot  of  this  measure  along  with  the  corresponding  statistical 
control  limits  that  are  derived  from  other  projects’  review  rates  at  those  same  points  in  the 
development  schedule.  The  X-axis  represents  the  time  at  which  the  designated  reviews 
occur,  while  the  Y-axis  is  the  review  rate.  The  calculated  review  rate  for  the  project  of 
interest  and  the  associated  quality  control  limits  are  plotted  on  this  graph.  As  can  be  seen 
from  the  graph,  at  the  third  major  review,  the  review  rate  is  higher  than  the  control  limits. 
This  indicates  that  not  enough  time  was  spent  in  reviewing  the  material.  Inadequate  time 
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spent  in  the  review  process  could  result  in  system  errors  going  undetected  until  later 
development  phases.  Defects  found  later  in  development  are  more  costly  to  fix  than  are 
defects  found  earlier  in  development,  and  this  measure  may  be  one  indicator  of  either  an 
increase  in  cost  to  fix  problems  found  late  in  development  or  a  decrease  in  quality  of  the 
end  product  (for  those  defects  that  go  completely  undetected). 


Review  Rate  (req/hr) 

80 

70 

60 

50 

40 

30 

20 

10 

0 

0-Jan  1-Jan  2-Jan  3-Jan  4-Jan 

Review  Date 


Figure  2:  Plot  of  Review  Rate 

7.2  Example  Product  Measure:  Defect  Density 

This  measure  reflects  the  weighted  average  number  of  major  defects  per  normalized 
system  engineering  “size”  of  a  given  project  within  an  organization.  A  “defect”  might  be 
defined  as  any  deviation  or  omission  from  the  system’s  requirements;  namely  an  error.  By 
major  defect  we  might  mean  one  that  would  result  in  the  system  not  being  able  to  perform 
its  mission  and/or  the  operation  of  the  system  could  result  in  potential  loss  of  human  life. 
Each  organization  would  need  to  define  its  own  “severity”  classification  and  then  decide 
upon  which  levels  of  that  scheme  constitute  major  errors.  Normalized  “size”  can  be 
measured  as:  number  of  requirements,  number  of  Function  Points,  number  of 
documentation  pages,  number  of  lines  of  computer  code  etc.,  normalized  by  an 
appropriate  factor.  The  important  factor  is  to  determine  a  common  measure  of  “size” 
within  the  organization  for  every  product.  The  information  needed  to  calculate  Defect 
Density  can  be  found  in  an  organization’s  problem  reporting  database  and  measurement 
database  (or  equivalent  database  that  keeps  the  “size”  of  the  project).  The  user  of  this 
measure  would  be  the  organization's  management,  the  various  project  team  leaders  and 
developers,  and  anyone  else  concerned  with  tracking  the  quality  of  the  organization  as  it 
pertains  to  product  quality.  This  measure  would  be  reported  on  at  least  an  annual  basis  to 
provide  the  overall  organization's  quality  trend.  This  measure  could  also  be  tracked  per 
project  and  reported  at  each  major  milestone  review. 

For  our  example,  suppose  we  consider  “size”  using  the  number  of  system  requirements 
with  a  normalizing  factor  of  100.  Suppose  we  have  3  projects  with  respective  weightings 
assigned  as  .5,  .25,  and  .25.  Management  feels  that  Project  1  (.5)  is  twice  as  important  as 


30 


the  other  two.  (Determining  weights  is  a  highly  subjective  judgment.  If  management  does 
not  specify  the  weights  to  employ,  equal  weighting  can  be  used.)  Suppose  for  the  last  four 
years  that  the  value  of  this  measure  has  been  reported  as  3.7,  3.4,  3.2,  and  3.1.  For  this 
year  the  cumulative  total  number  of  requirements  for  each  project  was  respectively  800, 
700,  and  300  respectively.  Normalizing  these  values  by  100  yields  8,  7,  and  3.  Suppose 
from  the  problem  report  database  we  find  the  cumulative  number  of  major  defects  for  each 
of  these  programs  were:  16,  4,  and  20  respectively.  Then  the  value  of  this  measure  for 
this  reporting  period  is  the  weighted  sum  of  the  normalized  defects  rate  per  100 
requirements: 


Defect  Density  =  (  .5  (16/8)  +  .25(4/7)  +  .25(20/3))  =  2.8 

The  organization  is  therefore  "averaging"  across  the  three  projects  about  2-3  major 
defects  in  every  100  system  requirements.  If  we  plot  this  measure  over  time  we  can  see 
how  well  the  organization  is  doing  in  improving  quality.  This  is  illustrated  in  Figure  3. 


1  2  3  4  5 


Figure  3:  Defect  Density  Plotted  Against  Year 

In  the  plot  we  see  the  organization's  goal  is  2  major  defects  for  every  100  system 
requirements.  The  organization  has  shown  steady  progress  to  achieve  that. 


7.3  Example  Progress  Measure:  Requirements  Stability 

This  is  a  measure  that  reflects  the  number  of  requirements  that  have  changed  (e.g.  added, 
modified,  or  deleted  from  the  last  baselining)  divided  by  the  current  number  of  baselined 
requirements.  The  higher  this  number  is  the  more  unstable  the  requirements  appear.  Low 
values  indicate  stability.  If  the  requirements  are  changing  too  fast,  this  introduces 
problems  in  all  areas  of  the  system  development,  especially  in  meeting  schedule  and 
keeping  within  budget.  The  data  for  this  measure  is  obtained  for  a  given  baseline  from  a 
requirements  baseline  management  database  or  equivalent.  This  measure  can  be  reported 
weekly  or  monthly  depending  upon  the  needs  of  the  organization  and  the  nature  of  the 
project.  This  measure  is  of  interest  to  project  management  and  the  system  developers. 


31 


For  an  example  of  this  measure  suppose  from  the  last  baseline:  5  new  requirements  were 
added,  6  deleted,  and  10  have  been  modified  for  a  total  of  21  changes.  At  the  current 
baseline  suppose  there  are  100  requirements.  Thus  the  requirements  stability  measure  is: 

Requirements  Stability  (present)  =  21/100  =  .21. 

Suppose  for  the  past  6  baselines  of  the  system,  the  values  were  1,  .6,  .5,  .3,  .27.  and  .18 
respectively.  (Notice  if  we  have  a  new  system  development  the  value  of  this  measure  is 
initially  1  since  we  have  all  new  requirements.)  Figure  4  shows  a  plot  of  this  measure 
against  the  baseline  version  number  with  baseline  0  being  the  start  of  this  new  system.  As 
we  can  see  from  this  plot  the  requirement  changes  are  slowing  down  indicating  the  number 
of  changes  is  getting  smaller.  We  see  that  we  have  achieved  our  goal  of  no  more  than  25 
requirement  changes  per  100  requirements  by  baseline  5.  Also  we  can  get  an  idea  of  how 
fast  we  are  achieving  this  stability  by  looking  at  the  slope  of  this  curve. 


Figure  4:  Plot  of  Requirements  Stability 
Vs  Baseline  Number 

The  user  of  this  measure  would  need  to  establish  the  acceptable  ranges  and  rates  of 
change.  In  addition  the  user  needs  to  be  aware  of  a  possible  ripple  effect  in  that  a  few 
modifications  to  major  requirements  could  affect  many  others,  thereby  inflating  this 
measure. 

The  importance  of  this  measure,  as  with  any  measure,  is  that  it  serves  as  an  indicator  or 
flag.  Something  has  changed  in  the  process.  That  change  may  be  good,  detrimental,  or 
neutral  to  our  system  development.  The  key  is  to  determine  why  the  value  of  the  measure 
has  changed,  make  a  determination  of  the  impact,  and  then  determine  whether  any  action 
is  required! 

The  reader  is  again  reminded  to  use  caution  in  drawing  inferences  in  any  of  the  above 
examples.  One  must  look  at  a  number  of  measures  to  get  a  total  perspective  of  the  system 
development  before  taking  any  action.  Consider  all  available  data  relating  to  the  problem 
both  qualitative  and  quantitative. 
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8.  Feedback  Forms 


Any  comments  or  suggestions  that  you  would  like  to  share  on  the  form  or  content  of  this 
Primer  are  welcome.  Please  feel  free  to  use  the  feedback  forms  provided  in  this  section  to 
communicate  your  thoughts.  (If  email  is  your  preferred  mode  of  communication,  the 
appropriate  email  address  to  send  your  comments  to  is:  garry.j.roedler@lmco.com) 

These  forms  can  either  be  faxed  to:  Garry  Roedler  (610)  531-1190  (USA),  or  mailed  to 
the  following  address: 

Garry  Roedler 

Lockheed  Martin  Management  and  Data  Systems 

PO  Box  8048 

Building  A,  Room  13A30 

Philadelphia,  PA  19101  USA 

If  you  have  any  further  questions,  please  call:  Garry  Roedler,  (610)  531-7845. 
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INCOSE  MWG  -  Measurement  Primer  Feedback  Form 

Name: _ 

Company: _ 

Address: _ 


Would  you  like  responses  to  your  comments  (Y/N)? _ 

Location  of  Comment  Description  of  Comment 

(Section,  Page  #) _ 


